Skip to main content

Andrew Saxe: Hallmarks of Deep Learning in the Brain

Movies Preview

texts
Andrew Saxe: Hallmarks of Deep Learning in the Brain


Published February 17, 2016
SHOW MORE


Talk by Andrew Saxe from Harvard University.  Given to the Redwood Center for Theoretical Neuroscience at UC Berkeley.

Abstract.
Anatomically, the brain is deep. To understand the ramifications of depth on learning in the brain requires a clear theory of deep learning. I develop the theory of gradient descent learning in deep linear neural networks, which gives exact quantitative answers to fundamental questions such as how learning speed scales with depth, how unsupervised pretraining speeds learning, and how internal representations change across a deep network. Several key hallmarks of deep learning are consistent with behavioral and neural observations. The theory can be further specialized for specific experimental paradigms. Taking visual perceptual learning as an example, I show that a deep learning theory accounts for neural tuning changes across the cortical hierarchy; and predicts behavioral performance transfer to untrained tasks as a function of task precision, restricted position training, and learning time. Together, these findings suggest that depth may be a key factor constraining learning dynamics in the brain. A better scientific understanding should eventually contribute to engineering advances, and I discuss one example from this work: a class of scaled, orthogonal initializations which permit rapid training of very deep nonlinear networks.


Language English
Collection opensource_media

comment
Reviews

There are no reviews yet. Be the first one to write a review.
SIMILAR ITEMS (based on metadata)
Community Media
texts
eye 2,259
favorite 0
comment 0
Community Media
texts
eye 50
favorite 0
comment 0
Community Media
texts
eye 228
favorite 0
comment 0
Community Media
texts
eye 144
favorite 0
comment 0
Community Media
by Abram Hindle, SkruntSkrunt
texts
eye 19
favorite 0
comment 0
Community Media
texts
eye 111
favorite 0
comment 0